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Abstract: To interface between the one-dimensional scanning of an image, and the application of a two- 
dimensional operator, an intermediate storage is required. For a square image of size n 2 , and a square 
operator of size m 2 , the minimum intermediate storage is shown to be n • (ra — 1). An interface of this size can 
be conveniently realized by using a serpentina delay line. New kinds of imagers would be required to reduce 
the size of the intermediate storage below n • (m — 1). 
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1. Introduction 

Image processing often requires the application of a two-dimensional operator over the image. Imaging 
devices usually scan the image along a one-dimensional path, and produce a stream of data points. In such 
cases, an intermediate storage is required, to interface between the one-dimensional scanning and the applica- 
tion of the two-dimensional operator. 

In this paper, the inter face problem will be examined. In particular, we shall determine the minimum 
size required of the intermediate storage. For a square image of size n 2 , and a square operator of size m 2 , the 
minimum intermediate storage is shown to be n • (m — 1). For small-size operators the intermediate storage 
may not add substantially to the overall size of the system. For example, a recent two-dimensional CCD 
scanner developed by Texas Instalments (Hall & Awtrcy, 1979) incorporates the intermediate storage required 
for a 3*3 operator on the scanner chip itself. For larger operators, however, the size of the intermediate 
storage becomes significant. For example, a recent CCD convolution chip constructed by Hughes is designed 
to support a 26 * 26 operator (Nudd et aL, 1979). It is intended to be applied to images of 1000 points on a 
side, in which case an intermediate storage of 25000 image points would be required. In such cases the size of 
'^e system is determined almost entirely by the intermediate storage requirement. Given die theoretical lower 
bound on the size of this storage, it is suggested that new kinds of imagers would be required for systems for 
which size is a dominant consideration. 



2. Using a so* pent km delay line of size n • (m — 1) 

A simple method for interfacing a one-dimensional scanner to a two-dimensional operator is shown in 
Figure 1. The camera e scans the image (assumed to be a square of n 2 points) row by row. The intensity values 
measured by the camera are fed into a serpentine delay line (S in Figure 1), consisting of m — 1 rows, each 
of length n. The serpentine delay line is tapped in m — l positions that are connected to the two-dimensional 
operator M (of size ra 2 ). In addition, there is a direct line from the camera to M, so that there are a total of m 
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input lines to M. 

After the first m — 1 rows have been scanned, they fill the intermediate storage S, and the first column 
of these rows resides in the rightmost column of S, ready to be delivered to M. The scanner then reads point 
(mj) in the image, and the first column of m points (point (1,1) to (m,l)) is transferred to M. Point (m,l) 
also enters the delay line, shifting each point in S by one position. The point in the lower-right corner of S 
is dropped from the memory and lost. Whenever a new point is now read by the scanner, a column of m 
points is transferred to M. After having scanned m points from the m th row, the entire upper-left square of m 2 
points resides in M, and the operator is applied to it. When the scanning of the entire image is completed, the 
operator has been applied to all the squares of size m 2 in the image. 

The intermediate storage S in this scheme is required to hold n[m— 1) image points. The next section 
will establish that this requirement cannot be reduced. Kvcn if one uses arbitrary scanning pattern (e.g., the 
image may be divided into sub-blocks, and each block scanned and operated upon separately), and arbitrary 
interconnections between S and M (i.e., M may have access to all the points in S ), the minimal intermediate 
storage would still be n • (m — 1). 
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Figure!: A serpentine 
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delay line S ( with m — I rows of length n each, interfaces a scanner C with (he 
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3. The minimal intermediate storage is n • (m — 1) 

Let the size of the intermediate storage be S. In this part we wish to show that S > n • (m — 1). 

Notation: We shall think of the image as a grid of n * n squares, n > 2. Each square in the grid is also 
called a point in the image. A two-dimensional operator of size m * m is a function of ra 2 arguments. The 
function is to be applied to every square of size m 2 points in the image. We shall assume that the size of the 
operator (m) is at most f . Finally, for a set P of points, ||P|| will denote the number of points in P. 

Definitions: Two distinct points in the image are called partners if they can be covered simultaneously by 
a square of size m 2 . That is, their distance in the image in either die horizontal or the vertical direction is 
at most m points. Similarly, the partners of a set P of points arc die partners of all the points in P, but not 
including the points in the set P itself. That is, 

partner s(P) = [J partner s(i) — P (1) 

UP 

When the scanner has read k points, k > S, some of the k points are no longer in die storage, and can no 
longer be retrieved. Let P be the set of points that have been scanned, but are no longer in the storage. It is not 
difficult to see tiiat at this stage the partners of the set P must reside in the intermediate storage. The reason is 
that if points p and q are partners, then there must exist a stage at which both p and q are in the intermediate 
storage. We can now obtain a lower bound on the size of the intermediate storage by estimating die number 
of partners of a set of k points. We shall establish the lower bound on the size of the intermediate storage by 
showing that for a certain choice of k, the number of partners is at least n{m — 1). 

Proposition 1: Let k be the smallest integer such diat k > *£. For diis choice of /c, any set of k points has 
at least n(m — 1) partners. 

Proof: By induction on m, die size of the two-dimensional operator. For m = 1 the claim holds, since 
die number of partners is zero. To prove the induction step we shall show that, in an image of size n 2 , when 
the size of the operator is increased from m = M to m = M + 1, (assuming M + 1 < § ), die set of partners 
is increased by at least n new points. To prove the induction step, we shall use the following two definitions. 



Definition I: Let I be the image, and Pa collection of points in die image. A neighbor of the set P is a 
point in I - P that shares an edge or a vertex on the grid with a point in P. 

Definition 2: In a grid of size n 2 , a set of k points is called admissible (and the grid itself is admissible) if : 

i n 2 <fc< 3 2 forn> 2 m 

4 4 



-n 2 <k<~n 2 forn=^2 
4 ~ 4 

Comment: Let P be the set of the k original points, and all their partners for an operator of size M. When 
the operator increases in size to M + 1, all the neighbors of P become new partners of the k original points. 
To prove proposition 1 it will be sufficient, therefore, to show that die set P has at least n neighbors. This will 
be the goal of proposition 2. 

Without loss of generality, we can assume that die set P is admissible. Since P contains the k original 

points, clearly f < ]\P\\. If ||P|| > i|n 2 , dien the k original points have at least \n 2 — k partners. 

"!M since k < ■*£ -+- 1, and § > M + 1, it would follow that die k original points have at least n • m 

partners. Proposition 1 would diercfore hold for die operator of size M + I, and die induction step would be 

unnecessary. 

Proposition 2: In a grid of n 2 points, any admissible set of K points has at least n neighbors. 

Pw# By induction on n, the size of die grid.^^ 
and in cither case there are at least 2 neighbors. We next assume thatproposition 2 holds for n — W, and 
prove it Torn — - N -J- L 

The general structure of the proof will be as follows. For convenience of reference, the K points will be 
called die black points, and the remaining points white. In these terms, we wish to show that in a grid of size 
(N + I) 2 any set of K points, |(N + I) 2 < K < \{N -f- I) 2 , has at leastN + I white neighbors/To use 
die induction hypodiesis we shall prove the folio wing claim: 
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Proposition 3: It is possible to remove a row and a column from the grid in such a way, that the resulting 
grid of size N 2 will be admissible. 

From the induction hypothesis, in the reduced grid die set of black points has at least N neighbors. The 
proof will be concluded by showing that when the missing row and column are re-introduced, the number of 
neighbors of the black set is increased by at least one. 

Proposition 3. 1: From an admissible grid of size (N + l) 2 it is possible to remove a row and a column, so 
that in the remaining grid of size N 2 the number of black points will be less than f N 2 . 

Proof: Let a/v+j denote the fraction of black points in the original grid. That is, the number K of 
black points is equal to Q/v-f 1 * [N + l) 2 . Similarly, a N is the fraction of black points in the reduced grid. In 
these terms, we assume in proposition 3.1 that \ < qn+\ < |, and wish to establish that a/v < f . 

Let Bij denote the total number of black points in row i and column j. Bij is the (number of black points 
in row i) + (the number of black points in row j). The difference between B^ and B^ is that if point (ij) is 
black, it is counted twice in Bij but only once in £ . Let B = L^w B iJ9 B = £^i\ B^. 

There are a N +\(N + l) 2 black points in the grid, and each one is counted 2(N + 1) times in 5, 
therefore: 

B-2(N + 1).q /v+1 (N + 1) 2 (3) 

In B each black point is counted one additional time, compared with J5, therefore: 

B = B-a N+1 (7V + l) 2 (4) 

There are (N + I) 2 terms B^ contributing to the sum B, dierefore the average contribution is: 

B =2a N+l (N + l)-a N+l (5) 



(N + l) 2 
Phcrc must exist a term B^ that contributes at least the average contribution. Proposition 3.1 will therefore be 



satisfied if: 



a N + { (N + l) 2 - (2a/v +l (N + 1) - a N+ i) < ^N 2 (6) 



If this inequality does not hold, then: 



* N +%{H + if - 1**n+£N 4- 1) - «n+i3 > jN a (7) 



But this implies a N+ i > f , which is a contradiction, since \ < a N +i < |. 1 

Proposition 3.2: In the conditions of proposition 3.1 it is possible to remove a row and a column so that in 
the reduced grid a/v > \ . 

The proof will be omitted, since it is entirely analogous to die proof of proposition 11. 

Comment: The row and column chosen in propositions 3.1 and 3.2 are not necessarily the same. If, in 
proposition 3.1, a N > |, or if, in proposition 3.2, a N < f , then proposition 3 is established. Proposition 
3 may still fail, however, to be satisfied if in proposition 3.1 a N < \ and in proposition 3.2 a N > \. It is 
required therefore to show that in this case we can still remove a row and a column such that in the reduced 
grid the set of black points is admissible. 

Let the row and column chosen in proposition 3.1 be row i and column j. The row and column chosen 
in proposition 3.2 will be denoted row it and column L Let us now remove from the grid row / and cdlumn /. 
Since the definition of an admissible set is slightly different for N — X we shall start with the case N > 3. In 
this case, proposition 3 would fail to be satisfied if either: 



or: 



a N+J (N + l) 2 -^>|iV 2 (9) 



Let us examine the first case. We may assume that 

^ +1 (/V + lf-ftr>^ 2 (10) 
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(sec the comment above). From (8) and (10) we conclude that: 

1 



Bu-B kl >-N 2 (11) 

Bit— B ki is the difference in the number of black points between rows / and jfc, after removing the I th column, 
hence B it — B ki < N. This, together with (11), implies N < 2, in contradiction with the assumption that 
N > 3. We conclude that for N > 3 (8) is impossible, and the inequality a N + { (N + l) 2 — B it > \N 2 
must hold. To include die case N = 2, a similar argument establishes that a N + { (N + l) 2 — Bu > \N 2 
mustholdforN> 2. 

The case in (9) can be handled analogously, yielding 

^ Bn-Bu>\N\ (12) 

which again cannot hold for N > 2. 

In conclusion, die fraction a N satisfies \ < a N < § for N > 3, and \ < a N < | for TV = 2. The 
set of black points in the reduced grid is therefore admissible. | 

Proposition 3 has established that a row and a column can be removed from die original grid of size 
(N + l) 2 , resulting in an admissible grid of size N. From die induction hypothesis, in the reduced grid the set 
of black points has at least N neighbors. To show that the set of neighbors in the original grid is larger (by at 
least one), we shall use the following claim: 

Proposition 4: proposition 3 can be satisfied by removing a row and a column that are not entirely black. 

Proof: It is sufficient to show that proposition 3.1 can be satisfied with a row and a column that contain 

at least one white point. We can assume without loss of generality that the original grid contains die maximal 

possible number of black points, since if proposition 3.1 can be satisfied (with a row and a column that are 

^^ not entirely black) for a maximal number of black points, it can also be satisfied for grids with fewer black 

points. For N -f 1 = 2m (an even number), the maximal number of black points is \ (N + l) 2 — 1. For 
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TV + 1 = 2m. -f 1, the maximal number is \{N + l) 2 - f : We shall examine the latter ease (the hrst can be 

established analogously). 

Let r be the number of completely black rows in the original grid, c the number of completely black 
columns. We can assume that r > 1, c > 1, since otherwise proposition 4 followcs directly from proposition 
3. Since Q N +i < |, either c < ££± orr < ^ must hold. Without loss of generality we shall assume 

thatc<^. 

r ITie r black rows and c black columns contribute a total of 

(r-fc)(N -1-1) -re (13) 

black points. The number of black points in the remaining rows and columns is therefore: 

3 i )2 _3 {r + c)(N + 1) _ j _ rc {14) 

4 4 

This number of black points is divided among N + 1 ■ — c columns, hence the average per column (denoted 

by A) is: 

a _ |(N + l) 2 -|-(r + e)(/V + l) + rc 

■ /4= ~" " ' N + k-c 

There must exist a column which contains at least r -f4 black points. By removing thiscolumn and one of the 
black rows we remove at least: 

(AT + l)+r-M — H (W) 

black points. It therefore remains to establish that: 

|(N + l) 2 -|-(iV + r + /t)<|/V 2 (IT) 



Inequality (17) reduces to: 
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p or c < i^+I inequality (18)holds, therefore proposition 4 is established § 
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Corollary: using the same argument, proposition 3.2 can be established with a row and a column that 
are not entirely white. Combining the two claims, it is straightforward to establish that proposition 3 can be 
satisfied by a row and a column that arc neither entirely black nor entirely white. 



Proposition 5: The set of black points in the original grid has at least N + 1 neighbors. 

Proof: From propositions 3 and 4, we can remove a row R and a column C (neither entirely black nor 
entirely white) from the original grid, and obtain an admissible grid. From the induction hypodiesis, the set of 
black points in the reduced grid has at least N neighbors. We have to show that by re-introducing R and C 
the number of neighbors increases by at least one. 

Since R and C contain both black and white points, they contain at least one "internal neighbor", i.e., a 
white point that borders a black point, where both points are in R \j C. We shall show that without the inter- 
nal neighbor the number of neighbors in the original grid is at least N, hence the total number of neighbors is 
at least TV + 1, as required. 

If a given point p is a neighbor of the black set in the reduced grid, it may or may not remain a neighbor 
when die missing row and column are re-introduced. Let us examine first the case where p is not a neighbor 
of the black set in the original grid. In this case p must be surrounded in the original grid by white points (see 
Figure 2). In this configuration, point q in R becomes a new neighbor, since at least one of the points a, 6, c 
must be black. Note that q cannot be an internal neighbor. (Comment: in Figure 2, point p borders the re- 
introduced column R. The argument is unchanged when it borders C instead of R. The case where p borders 
both R and C requires a slight extension of the argument, which will not be elaborated.) 

We conclude that there exists a one-to-one mapping from the neighbors in the reduced grid into the 
neighbors in die original grid, and tliat this mapping docs not include the internal neighbor. The number 
of neighbors in the grid of size N + 1 is therefore at least N + 1. This establishes die induction step in 
proposition 2, and concludes the proof of propositions 2 and 1. | 
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Figure 2: After column R has been re-introduccd, point p is no longer a neighbor of the black set. All the 
squares bordering p must therefore be white. Prior to the re-introduction of column 8 , p was a neighbor of 
the black set, hence at least one of die points a, b, c, must be black . Point q is therefore a neighbor of the black 
set. 
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4. Reducing the size of the intermediate storage 

The lower bound established in the preceding section implies that the intermediate storage may play a 
dominant role in determining the size of the system. The intermediate storage can, however, be reduced by 
scanning die image more than once. Suppose, for example, that s rows are scanned in parallel. The first scan- 
path scans rows 1 to s, the second scans rows 2 to s + 1, etc. Each image point is scanned in this scheme s 
times. Using the serpentine delay line of Section 2, it is possible now to connect s lines directly to the two- 
dimensional operator M. The intermediate storage S will then have only (m — s) rows. This scheme gives 
a simple (though not optimal, in terms of the storage size) method of reducing the size of the intermediate 
storage to n * (m — - s), where s is the number of times each point is scanned. In particular, by having s = m 
the intermediate storage becomes unnecessary. 
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